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ABSTRACT 


The  purpose  of  this  study  is  to  evaluate  response  surface  methodology  as  a 
sensitivity  analysis  tool  in  the  area  of  decision  analysis.  The  advent  of  low-cost  personal 
computer  software,  such  as  DPL™^  has  created  an  accessible  tool  with  the  ability  to 
frame  and  solve  influence  diagrams  for  decision  problems.  The  limited  sensitivity  analysis 
capabilities  used  in  this  field  has  created  a  need  for  improvement. 

This  study  provides  a  comparison  of  current  sensitivity  analysis  techniques  vs  those 
made  possible  through  response  surface  methodology  (RSM).  Sensitivity  analysis 
alternatives  are  demonstrated  on  a  decision  problem  concerning  the  evaluation  of  force 
structure  options  for  the  Department  of  Defense.  Sensitivity  analysis  is  performed  on  both 
one-way  and  two-way  perturbations  of  input  variables. 

The  most  significant  contribution  of  this  thesis  is  the  effectiveness  of  RSM 
techniques  for  the  sensitivity  analysis  of  a  decision  problem.  RSM  is  proven  to  be  an 
effective  and  highly  efficient  approach.  The  combination  of  two  analysis  tools,  the 
influence  diagram  solver  and  the  RSM  sensitivity  analysis,  has  effectively  saved  the 
decision-maker  valuable  resources  and  increased  the  information  made  available.  The 
resources  saved  are  analysis  and  computer  time.  While  the  increased  information  includes 
multiple  factor  sensitivity  analysis,  as  well  as  additional  insights  gained  from  the 
coefficients  of  the  RSM  equations.  This  capability  will  enhance  the  analysis  of 
uncertainty  in  decision  problems. 
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RESPONSE  SURFACE  METHODOLOGY  AS  A  SENSITIVITY 
TOOL  IN  DECISION  ANALYSIS 


1.  Introduction 


Background 

Decision  makers  in  the  Air  Force  are  interested  in  an  increased  ability  to  evaluate 
uncertainty  in  decision  analysis.  Evaluating  the  possible  alternatives  is  important,  but  so  is 
analyzing  the  impact  of  uncertainty  in  the  input  variables.  The  decision  maker  wants  to 
know  what  will  happen  if  current  inputs  are  off  the  mark.  What  if  the  threat  to  national 
survival  has  a  90%  chance  of  being  high  in  the  next  20  years  instead  of  the  planned  70%  ? 
What  if  the  cost  of  a  buildup  in  force  structure  is  higher  than  expected?  Will  this  impact 
the  optimal  decision  policy?  In  order  to  answer  these  questions  with  some  level  of 
confidence,  the  analyst  must  have  the  appropriate  tools  available.  These  tools  should 
permit  the  analyst  to  recommend  a  course  of  action  and  provide  information  on  the 
sensitivity  of  his  recommendation  to  changes  in  the  input  parameters.  This  sensitivity 
information  could  involve  changing  a  number  of  input  parameters  at  the  same  time,  which 
exceeds  the  current  capability  of  decision  analysis  tools.  The  topic  of  this  paper  is  a 
combination  of  methods,  joining  decision  analysis  and  response  surface  methodology, 
which  shows  the  potential  to  meet  this  need. 

There  are  many  types  of  approaches  and  models  to  analyze  Air  Force  problems. 

The  goal  is  to  give  the  decision-maker  dependable  insight  on  the  options.  First,  the  analyst 
and  decision-maker  must  agree  on  the  problem.  It  is  of  utmost  importance  that  the 
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presentation  is  clear  and  to  the  point.  This  is  where  a  good  understandable  graph  or 
picture  of  the  information  is  critical. 

One  modeling  technique  that  captures  and  presents  the  key  information  is  the 
influence  diagram.  Shachter,  who  created  an  algorithm  to  solve  the  influence  diagram, 
defines  them  as  follows:  "An  Influence  Diagram  is  a  graphical  structure  for  modeling 
uncertain  variables  and  decisions  and  explicitly  revealing  probabilistic  dependencies  and  the 
flow  of  information."  (7:871).  The  influence  diagram  has  the  advantage  of  graphically 
depicting  a  problem  and  how  the  variables  are  related;  all  in  one  picture.  This  is  good  not 
only  from  an  analyst's  view,  but  also  for  the  decision-maker,  who  can  verify  that  the  correct 
problem  is  being  addressed. 

Recently,  there  has  been  much  work  by  software  developers  to  produce  a  viable 
package  to  solve  influence  diagrams.  One  such  package  is  Decision  Programming 
Language  (DPL™).  It  gives  the  analyst  a  straightforward  tool  to  model  problems  using 
influence  diagrams  and  then  solve  them. 

One  of  the  limitations  encountered  using  this  software  is  the  availability  of  only 
limited  sensitivity  analysis  tools.  The  software  includes  an  option  to  do  deterministic  one¬ 
way  sensitivity  analysis,  but  not  any  real  two  or  more-way  analysis.  It  is  in  this  general 
area  of  sensitivity  analysis  that  more  work  is  needed.  In  this  paper.  Response  Surface 
Methodology  (RSM)  is  utilized  as  a  sensitivity  tool  for  the  analysis  of  a  problem  solved 
with  a  influence  diagram. 

Research  Objective 

The  objective  of  this  research  is  to  show  that  the  use  of  response  surface 
methodology  can  facilitate  the  constmction  of  effective  and  efficient  sensitivity  analysis  in 
the  field  of  decision  analysis.  Effectiveness  (to  within  an  acceptable  tolerance  of  error)  is 
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measured  against  current  methods  .  Efficiency  is  evaluated  through  resource  (analysis  and 
computer  time)  utilization  as  compared  to  standard  practice. 

Benefits 

There  are  a  number  of  benefits  from  using  response  surface  methodology  for 
sensitivity  analysis  on  a  decision  problem.  The  first  is  a  substantial  savings  in  the  total 
number  of  model  runs.  As  the  model  increases  in  complexity,  the  savings  are  tremendous. 
Subsequently,  precious  time  to  produce  sensitivity  information  is  salvaged.  This  time 
reduction  is  quite  often  the  most  significant  from  the  decision-maker's  point  of  view.  The 
decision-maker  receives  the  benefit  of  time-critical  information  as  well  as  the  added  insight 
made  available  from  the  use  of  RSM.  The  additional  information  gained  from  the  use  of 
RSM  is  due  to  the  extension  of  current  analysis  options,  such  as:  the  estimation  of  specific 
coefficients  (effects)  of  significant  variables,  expanding  the  one-way  sensitivity  analysis  to 
two-way  and  beyond,  and  the  ability  to  track  multiple  factor  interactions.  The  cost  of  these 
benefits  is  a  limited  loss  of  accuracy,  well  within  the  tolerances  required  for  sensitivity 
analysis. 

Scope 

This  thesis  shows  that  RSM  can  be  practically  applied  in  the  sensitivity  analysis  of  a 
complex  problem  posed  as  an  influence  diagram.  The  measure  of  efficiency  is  subjectively 
evaluated,  in  a  general  compaiison  of  RSM  vs  standard  sensitivity  analysis  results.  The 
sensitivity  analysis  is  accomplished  for  both  one-way  and  two-way  perturbations  of 
selected  factors.  This  analysis  is  done  to  identify  critical  input  variables  and  evaluate 
perturbations  of  their  value,  as  well  as  perturbations  of  the  their  probability  distributions,  as 
they  impact  on  the  overall  measurement  of  the  solution.  These  input  variables  are  subject 
to  some  uncertainty  that  is  out  of  the  control  of  the  decision  maker.  The  focus  of  the  thesis 
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is  not  to  solve  the  problem  being  modeled  but  to  evaluate  the  process  itself.  With  this  in 
mind,  this  paper  presents  an  analysis  which  can  be  easily  communicated  to  the  decision 
maker  without  a  need  to  understand  all  of  the  complex  methods  involved. 

Assumptions 

1.  The  response  surface  generated  by  the  model  can  be  accurately  portrayed  and 
analyzed  using  first  or  second  order  approximations  over  a  small  region  of  interest. 

2.  The  primary  focus  of  the  decision-maker  is  to  understand  the  underlying 
uncertainty  involved  in  making  the  decision. 

Results 

This  thesis  shows  that  using  RSM  to  do  sensitivity  analysis  on  problems  solved  with 
influence  diagrams  is  a  sensible  approach.  This  methodology  substantially  increases  the 
explanatory  power  of  sensitivity  analysis  on  decision  problems.  The  approach  utilized 
combines  two  proven  analysis  techniques  to  offer  the  analyst  another  tool  and  present  the 
decision  maker  a  more  beneficial  presentation.  The  results  indicate  that  complex  decision 
problems  can  be  analyzed  with  a  substantial  savings  in  analyst  man-hours  and  computer 
resources.  The  savings  are  a  result  of  the  utilization  of  RSM  to  estimate  the  response 
surface  generated  by  solving  the  influence  diagram.  The  subsequent  predictive  equations 
are  highly  accurate  estimations,  which  are  used  to  produce  sensitivity  information  for 
various  perturbations  in  the  input  variables. 

Presentation  of  Results 

Chapter  II  provides  background  on  the  fundamental  tools  in  decision  analysis.  This 
includes  information  on  decision  trees,  influence  diagrams  and  solution  capabilities.  This 
chapter  also  includes  the  current  framework  of  sensitivity  analysis  and  some  of  the 
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possibilities  of  the  RSM  metamodelling  paradigm.  This  chapter  builds  the  foundation  for 
the  methodology  that  is  explained  in  Chapter  III.  Chapter  III  describes  the  model  used,  as 
well  as  providing  a  road  map  for  the  sensitivity  analysis.  This  includes  the  standard 
sensitivity  analysis  offered  by  DPL™^  several  possible  two-way  sensitivity  techniques,  and 
the  chronology  of  the  response  surface  methodology.  Next,  an  analysis  of  the  results  is 
presented  in  Chapter  IV.  The  analysis  includes  the  comparison  of  RSM  to  the  other 
techniques  used.  Finally,  Chapter  V  provides  a  summaiy  of  the  conclusions,  along  with 
recommendations  for  further  utilization  of  RSM  in  decision  analysis. 
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II.  Background 


Decision  Analysis 

Howard  and  Matheson  describe  the  job  of  the  decision  analyst  "...as  that  of 
making  'opaque'  situations  'transparent,'  so  that  the  [decision-maker]  clearly  sees  what  to 
do"  (4:14).  The  overall  goal  of  the  decision  analyst  must  be  examined  to  evaluate  the 
various  tools  of  the  trade.  The  analyst  should  construct  a  model  that  shows  how  all  the 
variables  are  related  and  the  outcomes  of  various  decisions.  It  is  critical  to  find  the 
solution,  but  also  to  answer  the  correct  question  and  clearly  convey  that  answer  to  the 
decision  maker.  Clearly  identifying  the  problem  is  an  integral  part  of  the  analysis  and  is 
referred  to  as  framing  the  problem.  Framing  is  frequently  the  hardest  part  of  the  problem. 
The  analyst  has  to  identify  the  options  available,  the  uncertain  variables  that  relate  to 
outcomes,  and  decide  how  to  model  the  uncertain  variables.  Further,  the  relationship 
between  these  uncertain  variables  and  the  value  must  be  determined.  The  analyst  must 
select  a  tool  that  is  capable  of  solving  the  problem  and  flexible  enough  to  effectively 
communicate  with  the  decision-maker. 

Originally,  the  tool  used  most  frequently  to  display  and  solve  a  decision  problem 
was  the  decision  tree  (see  Figure  2.1).  The  decision  tree  is  useful  for  asymmetric 
problems  with  relatively  few  variables.  This  tool  can  be  used  to  solve  very  complex 
problems,  but  these  problems  can  turn  the  tree  into  a  complex  maze  of  branches  with 
associated  informational  details.  This  disadvantage  is  not  insunnountable  for  the  analyst, 
but  can  be  for  the  decision  maker.  The  decision  maker  has  to  digest  this  large  diagram 
and  evaluate  whether  it  portrays  the  decision  at  hand.  Remember,  the  decision  maker  may 
know  little  about  the  analyst's  tools.  For  this  reason,  influence  diagrams  have  been  used  to 
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portray  the  analyst's  understanding  of  the  problem.  The  calculations  can  be  accomplished 
separately  with  the  equivalent  decision  tree  or  influence  diagram  solver. 


Figure  2.1  Decision  Tree 


Influence  Diagrams.  In  an  influence  diagram,  the  problem  situation  is 
condensed  to  the  decisions,  the  important  unceitain  variables,  and  the  values  (see  Figure 
2.2).  These  three  node  types  are  tied  together  with  arcs  to  model  the  problem  without  all 
of  the  duplication  of  a  decision  tree.  The  goal  is  to  use  the  influence  diagram  to 
communicate  with  the  decision  maker  and  use  an  underlying  decision  tree  or  influence 
diagram  algorithm  to  do  the  analysis. 

In  1986,  Shachter  proposed  an  algorithm  for  evaluating  influence  diagrams 
(7:871-872).  This  algorithm  enabled  the  influence  diagram  to  be  solved  directly. 
Shachter  describes  the  influence  diagram  as, "...  intuitive  enough  to  communicate  with 
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decision  makers  and  experts  and,  at  the  same  time,  precise  enough  for  normative  analysis" 
(7:871).  The  capability  to  solve  the  influence  diagram  was  an  important  step  for  decision 
analysis.  This  capability  provides  analysts  with  a  good  communication  tool  in  a 
convenient  structure  for  computer  solution  techniques  (8:603). 


Figure  2.2  Influence  Diagram 


However,  this  technique  has  not  been  used  much  outside  of  academic  circles  until  recently. 
Decision  Programming  Language  (DPL™)  has  changed  that  with  the  ability  to  solve 
influence  diagrams  via  a  low  cost  microcomputer  software  package.  DPL™  solves  the 
influence  diagram  by  using  an  influence  diagram  algorithm  to  do  the  calculations  (1).  This 
allows  it  to  accomplish  deterministic  sensitivity  analysis  on  one  variable  at  a  time.  The 
output  includes  cumulative  risk  profiles  of  the  alternatives,  rainbow  diagrams,  and  tornado 
diagrams.  A  general  limitation  of  the  technique  (and  the  software)  is  the  inability  to  do  in- 
depth  (two-way  or  higher)  sensitivity  analysis. 


2-3 


Sensitivity  Analysis 

Although  the  sensitivity  of  any  one  variable  can  be  an  effective  screening  tool  for 
many  problems,  stopping  here  would  leave  out  a  large  part  of  the  potential  information 
gained  from  a  model.  If  the  problem  contains  many  variables,  what  is  the  effect  of  two  or 
three  being  slightly  different  than  anticipated?  If  the  answer  doesn't  change  the 
recommended  action,  it  doesn't  matter.  If  it  does  change  the  recommendation,  the 
decision  maker  needs  to  know.  One  feasible  approach  would  be  to  exhaustively 
enumerate  the  possibilities  and  run  the  model  at  all  the  possible  combinations  of  variable 
settings.  This  is  reasonable  for  small  problems  with  few  variables,  but  becomes  a 
exponentially  large  process  with  increased  problem  size.  An  approach  that  significantly 
reduced  the  computational  complexity  was  suggested  by  Smallwood  and  Morris  (9:61- 
80).  Smallwood  and  Morris  used  one-way  sensitivity  analysis  to  screen  the  critical 
random  variables  from  the  rest  of  the  field.  They  realized  that  to  run  a  full  factorial  design 
of  the  nine  variables  at  three  levels  would  take  3^  or  19683  individual  runs  of  the  model. 
This  would  take  up  a  costly  amount  of  analyst  time  and  computer  resources,  so  they 
decided  to  group  the  nine  critical  state  variables  into  three  classes.  They  developed  a 
worst  case  scenario  for  each  class  of  variables  which  defined  eight  global  scenarios  (9:61- 
80).  Running  these  global  scenarios  proved  to  be  a  effective  analysis  tool  for  generating 
strategy  regions,  but  condensing  the  variables  requires  general  assumptions  about  the 
groupings  and  combined  effects.  This  technique  provides  additional  information  to  the 
decision  maker,  but  limits  the  ability  to  trace  variable  effects  beyond  the  variable  group. 

Response  Surface  Methodology 

A  different  approach  was  used  by  Stone,  who  studied  the  non-conformity  of  parts 
in  the  Air  Force  Supply  System  (10).  A  simple  model  was  developed  for  determining  the 
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best  policy  for  sampling  parts  in  the  inventory.  The  model  was  displayed  as  an  influence 
diagram  and  solved  using  INDIA.  The  sensitivity  analysis  was  done  using  response 
surface  methodology.  Stone  found  that "  RSM  [Response  surface  methodology]  provides 
for  a  better  sensitivity  analysis  because  RSM  explores  the  interaction  effects  between  the 
variables  in  a  more  structured,  consistent,  and  reliable  way"  (10:2).  Stone's  thesis 
provides  initial  groundwork  for  applying  response  surface  methodology  to  decision 
analysis  problems. 

Complex  models  are  frequently  used  in  the  field  of  simulation.  Large  Air  Force 
simulation  models  take  hours  to  make  one  run,  so  common  techniques  like  multiple-run 
sensitivity  analysis  are  difficult  to  perform.  Response  surface  methodology  has  been  used 
to  do  sensitivity  analysis  on  many  of  these  simulation  models.  These  models  have  many 
variables  and  usually  only  a  fraction  of  them  are  critical  over  a  small  design  region.  Meidt 
and  Bauer  take  advantage  of  this  property  and  use  response  surface  methodology  to  create 
metamodels  for  simulation  (see  Figure  2.3)  (6:183-191). 

In  their  article,  Meidt  and  Bauer  describe  the  process  of  group  screening  as 
follows:  "Group  screening  combines  sets  of  factors  that  are  believed  to  exhibit  like 
behavior  in  the  model.  Each  group  is  treated  as  a  single  variable  and  regressed  against  the 
responses  using  a  low  resolution  design"  (6:184).  Group  screening  is  similar  to  single 
factor  screening  in  that  it  can  be  accomplished  with  a  low  resolution  design  to  narrow  the 
field.  This  technique  would  be  extremely  beneficial  where  the  uncertain  variables  are 
closely  correlated  to  the  response  in  a  large  model.  It  would  drastically  reduce  the  number 
of  runs  required  while  validating  the  lumping  of  variables  where  the  groups  proved  to  be 
insignificant.  Such  reduction  would  have  an  application  in  the  sensitivity  analysis  of  many 
complex  decision  analysis  problems. 
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Once  the  group  screening  nan'ows  the  field  to  critical  groups  of  input  variables, 
another  low  resolution  design  (resolution  III)  is  run  on  the  main  factors  remaining.  From 
here,  a  first  order  (linear)  model  can  be  used  to  estimate  the  response  surface  over  a  small 
area  of  interest.  This  model  is  regressed  and  tested  for  lack  of  fit.  If  the  model  proves  to 


GROUP  1 

GROUP  2 

GROUPS 

GROUP  1 

GROUPS 


GROUP  SCREENING 

- 1 - 

1 

FACTOR  SCREENING  , 

Figure  2.3  RSM  Metamodelling  Paradigm 
*  Adapted  from  (6:184) 


be  significant,  then  the  method  of  steepest  ascent  can  be  used  as  a  gradient  search 
technique.  At  some  point,  there  may  be  a  lack  of  fit  and  a  second  order  model  can  be  used 
to  better  fit  the  surface  and  eventually  locate  local  optimum.  This  second  order  model 
requires  a  higher  resolution  (resolution  V  or  better)  design  to  include  the  interaction 
terms.  Using  this  methodology,  valuable  insight  is  gained  into  the  response  function  over 
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a  range  of  inputs.  This  technique  can  not  only  be  used  for  optimization,  but  also  for 
sensitivity  analysis  on  the  predicted  surface. 
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III.  Methodology 


Problem  Description 

Air  Force  decision  makers  are  greatly  interested  in  expanding  the  ability  to 
evaluate  uncertainty  in  modeling.  This  ability  requires  the  analysis  of  unceitainty  in  the 
model  parameters,  including  subjective  value  and  probability  inputs,  and  presenting  the 
results  in  a  understandable  format. 

Here,  a  decision  analysis  model  is  developed  to  examine  the  major  uncertainties  in 
assessing  force  requirements.  The  DOD  is  significantly  restmcturing  U.S.  military  forces. 
Until  recently,  the  forces  were  designed  to  fight  and  win  two  major  regional  contingencies 
(MRC).  Due  to  significant  budget  constraints,  the  U.S.  must  consider  military  force 
structures  with  lower  capabilities.  However,  significant  uncertainties  exist  about  future 
threats  to  national  survival  and  national  interests.  Suppose  there  are  three  force  structure 
alternatives  under  consideration:  two  MRC  force,  one  MRC  force,  and  one-half  MRC 
force.  The  threat  to  national  survival  and  national  interests  over  the  next  20  years  can  be 
grouped  into  three  categories:  high  threat,  medium  threat,  and  low  threat.  National 
interests  include  threats  to  countries  that  have  international  security  agreements  with  the 
U.S.  Suppose  the  following  probabilities  are  obtained  from  intelligence  estimates: 


Threat 

Threat  to  National  Survival 

Threat  to  National  Interests 

High 

0.1 

0.3 

Medium 

0.2 

0.6 

Low 

0.7 

0.1 

Table  3.1  Probability  of  Threats  to  National  Survival  and  National  Interests 
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Next,  the  intelligence  data  on  the  expected  threat  is  included.  The  predictions  are 
based  on  the  assumption  that  the  intelligence  community  has  a  higher  probability  of 
correctly  predicting  threats  to  national  survival  than  to  national  interests.  The  following 
probabilities  are  for  the  intelligence  prediction  given  the  actual  threat  to  national  survival: 


Intelligence  Prediction  \ 

Threat  to  National 

Survival 

High 

Medium 

Low 

High 

0.8 

0.2 

0.0 

Medium 

0.2 

0.6 

0.4 

Low 

0.0 

0.2 

0.6 

Table  3.2  Intelligence  Prediction  given  Threat  to  National  Survival 


The  following  probabilities  are  for  the  intelligence  prediction  given  the  actual  threat  to 
national  interests; 


Intelligence  Prediction  \ 

Threat  to  National 

Interests 

High 

Medium 

Low 

High 

0.7 

0.3 

0.2 

Medium 

0.2 

0.4 

0.3 

Low 

O.I 

0.3 

0.5 

Table  3.3  Intelligence  Prediction  given  Threat  to  National  Interests 
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National  objectives  are  to  insure  national  survival  and  to  defend  national  interests. 
The  possible  outcomes  of  National  Sumval  and  National  Interests  are: 

•  Secure 

•  Marginal 

•  Insecure 

The  probabilities  of  the  outcomes  of  National  Survival  and  National  Interests  depend  on 
the  force  structure  decision  and  the  threat.  The  following  table  provides  the  outcomes  and 
probabilities: 


Force  Structure 

Threat 

National  Survival  / 

National  Interests 

Two  MRC 

All 

Secure,  1.0 

One  MRC 

High 

Marginal,  0.5 

Insecure,  0.5 

One  MRC 

Medium,  Low 

Secure,  1.0 

One-half  MRC 

High 

Insecure,  1.0 

One-half  MRC 

Medium 

Marginal,  0.5 

Insecure,  0.5 

One-half  MRC 

Low 

Secure,  1.0 

Table  3.4  Probability  of  outcomes  given  Force  Structure  and  Threat 


There  are  two  decisions  to  make;  the  first  is  the  initial  force  structure,  and  when 
the  threat  is  updated,  the  level  of  force  buildup.  The  force  buildup  has  the  same  options 
(1/2,  1,  or  2  MRC)  given  sufficient  lead  time.  The  value  of  possible  outcomes  is 
measured  with  an  additive  utility  function  consisting  of  three  individual  utilities.  These 
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individual  utilities  are  weighted  according  to  national  objectives.  The  overall  utility 
function  looks  like  this: 

U(Survival,  Interests,  Cost)  =  a  *  U(Survival)  +  b  *  U(Interests)  +  c  *  U(Cost) 

The  weights  of  the  three  utilities  must  sum  to  one  to  meet  the  independence  criteria.  The 
utilities  are  assigned  so  that  a  value  of  one  is  the  best  outcome.  Hence,  we  are  striving  to 
maximize  utility.  The  individual  utilities  range  from  zero  to  one  and  are  assigned 
according  to  the  outcomes  shown  in  Tables  3.5  and  3.6. 


Outcome 

National  Survival  Utility 

National  Interests  Utility 

Secure 

1.0 

1.0 

Marginal 

0.5 

0.5 

Insecure 

0.0 

0.0 

Table  3.5  Utility  of  National  Survival  and  National  Interests  given  Outcome 


Initial  Force  Structure 

Force  Structure  Buildup 

Force  Structure  Cost  Utility 

Two  MRC 

No  Change 

0.0 

One  MRC 

Two  MRC 

0.2 

One  MRC 

No  Change 

0.4 

One-half  MRC 

Two  MRC 

0.5 

One-half  MRC 

One  MRC 

0.7 

One-half  MRC 

No  Change 

1.0 

Table  3.6  Force  Structure  Cost  Utility  given  Initial  and  Buildup  Force  Structure 
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Unfortunately,  if  we  decide  to  build  up  our  force  structure,  the  buildup  may  not  be 
complete  in  time  to  meet  the  threat.  Therefore,  this  uncertainty  is  modeled  with  a  force 
available  variable.  The  assumed  probabilities  are  found  in  Table  3.7. 


Initial  Force 

Force  Structure 

Two  MRC 

One  MRC 

One-half  MRC 

Structure 

Buildup 

Two  MRC 

1.0 

0.0 

0.0 

One  MRC 

Two  MRC 

0.5 

0.5 

0.0 

One  MRC 

No  Change 

0.0 

1.0 

0.0 

One-half  MRC 

Two  MRC 

0.125 

0.375 

0.5 

One-half  MRC 

One  MRC 

0.0 

0.5 

0.5 

One-half  MRC 

0.0 

0.0 

1.0 

Table  3.7  Probability  that  the  Force  Structure  is  available  to  meet  Threat 


Model  Description 

Understanding  the  Influence  Diagram.  The  objective  of  this  model  is  to 
optimize  the  utility  of  the  force  structure  decision.  The  initial  force  structure  decision 
includes  the  three  force  structure  alternatives  previously  discussed,  but  are  not  displayed  in 
the  influence  diagram.  This  is  where  the  difference  between  a  decision  tree  and  a  influence 
diagram  becomes  apparent.  The  influence  diagram  depicts  a  macroscopic  view  of  the 
problem  with  emphasis  on  how  the  variables  are  related.  What  is  not  seen  is  the 
underlying  branches  of  alternatives  and  probabilities  that  have  been  defined  in  the  framing 
process.  This  high-resolution  information  is  hidden,  while  the  critical  relationships  among 
the  variables  are  shown  as  arcs.  These  arcs  define  when  information  is  known  relative  to 
decisions  and  whether  variables  are  probabilisticly  dependent  on  one  another. 
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Uncertain  Force  Structure 


Figure  3.1  Influence  Diagram  of  Force  Structure  Decisions 


The  influence  diagram  in  Figure  3. 1  shows  that  the  initial  force  structure  decision 
will  affect  the  force  structure  buildup  decision,  the  force  structure  available,  and  the  force 
structure  cost.  It  is  sensible  that  the  initial  force  structure  will  determine  the  options  for  a 
force  structure  buildup,  as  well  as  the  overall  cost  of  the  force.  The  force  structure  cost, 
defined  with  a  utility  ranging  from  zero  to  one,  is  determined  by  the  force  structure  chosen 
and  is  represented  with  a  value  node.  The  value  node  has  no  probabilistic  branches.  The 
value  node  infoimation  is  deterministic  once  the  inputs  are  defined.  This  node's 
information  is  then  used  as  an  input  to  our  overall  utility  function  contained  in  the  final 
value  node.  These  relationships  of  one  node  to  another  outline  the  fundamental  structure 
of  the  model.  This  structure  is  simply  explained  in  the  influence  diagram  providing  a 
powerful  communication  tool. 
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Input  Variables.  The  model  contains  seven  uncertain  variables  displayed  as 
ovals  in  the  influence  diagram.  These  uncertain  variables  are  derived  from  the  tables 
found  previously  in  the  problem  description.  These  variables  are  listed  below: 

National  Survival  threat  Intelligence  prediction  -  the  probability  of 
intelligence  predicting  future  national  suiwival  threats  given  the  actual 
threat. 

National  Survival  Threat  -  the  probability  of  the  actual  national  survival 
threat  being  either  high,  medium,  or  low. 

National  Survival  -  the  US  national  survival  probability  and  utility  of 
being  secure,  marginally  secure,  or  insecure  dependent  on  the  actual 
national  survival  threat  and  force  structure  available. 

Force  Structure  Available  -  the  probability  of  the  force  structure  attaining 
the  desired  buildup  level  given  the  initial  force  structure. 

National  Interests  threat  Intelligence  prediction  -  the  probability  of 
intelligence  predicting  future  national  interests  threats  given  the  actual 
threat. 

National  Interests  Threat  -  the  probability  of  the  actual  national  interests 
threat  being  either  high,  medium,  or  low. 

National  Interests  -  the  US  national  interests  probability  and  utility  of 
being  secure,  marginally  secure,  or  insecure  dependent  on  the  actual 
national  interests  threat  and  force  structure  available. 

The  remaining  nodes  are  for  the  force  strticture  cost,  the  overall  utility  function, 
and  the  cost  weight  parameter.  The  force  structure  cost  node  identifies  the  utility  of  all 
the  possible  combinations  of  initial  force  structure  and  force  structure  changes.  The 
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overall  utility  function  is  the  sum  of  the  three  individual  utilities  weighted  by  the  cost 
weight  parameter  as  seen  in  Equation  (3-2). 

U(Survival Interests, Cost)  =  0. 5  *  U(Survival)  +  0. 25  *  U(lnterests)  +  0.25* U(Cost)  (3.2) 

This  cost  weight  was  initially  set  to  assert  that  the  national  survival  utility  is  twice  as 
important  as  the  national  interests  or  force  stmcture  cost  utilities. 

Output  Variables.  The  model's  one  output  is  the  maximum  expected  utility  of 
the  optimal  decisions.  This  includes  the  expected  utility  ranging  from  zero  to  one,  but  also 
the  decision  policy  that  would  yield  this  optimal  number.  There  are  six  possible  decision 
policies  including  the  force  structure  buildup  options.  The  optimal  expected  utility  is  a 
maximum  of  the  three  initial  force  structure  alternative's  expected  utilities.  This  fact  can 
be  used  to  break  the  output  down  into  three  separate  responses.  These  three  response 
surfaces  are  evaluated  and  discussed  in  the  RSM  sensitivity  analysis  section. 

Sensitivity  Analysis  using  DPLTM 

Grouping  Variables  and  Initial  Perturbances.  Initially  the  variables  were 
placed  into  three  main  groups.  All  of  the  variables  which  were  probabilities,  such  as  the 
probability  of  the  threat  to  national  survival  being  high,  were  grouped  as  probability 
variables.  The  probability  vaiiable  group  included  51  individual  variables.  The  utility 
variable  group  included  12  variables,  such  as  the  force  stimcture  cost  utility  of  a  one  MRC 
initial  force  with  a  buildup  to  a  two  MRC  force.  The  last  group  is  the  utility  weight 
group,  which  includes  the  three  weighted  coefficients  in  the  overall  utility  function.  The 
perturbations  of  these  various  groups  are  limited  by  the  relationships  in  each. 
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The  probabilities  of  the  various  outcomes  must  obviously  sum  to  one  and  therefore 
had  to  be  perturbed  in  much  the  same  way  as  the  utility  weights.  This  vector  of 
probabilities  is  varied  by  changing  the  main  probability,  and  altering  the  other  two 
elements  of  the  set  by  a  negative  one-half  the  change.  The  individual  main  probabilities 
are  all  varied  by  plus  or  minus  0.1  about  their  initial  value.  An  example  of  how  the 
National  Survival  threat  probabilities  are  perturbed  is  shown  in  Table  3.8. 


Threat 

Threat  to  National  Suiwival 

Perturbation  of  probability  vector 

High 

0.1 

0.1+  ProbabilityPerturbation 

Medium 

0.2 

0.2  -  (0.5*ProbabilityPerturbation) 

Low 

0.7 

0.7  -  (0.5*ProbabilityPerturbation) 

Table  3.8  Perturbation  of  National  Survival  Threat  Probability  (NSTP) 


The  utility  variable  group  contains  the  various  utility  weights  which  are  assessed  by 
the  decision  maker  for  each  possible  outcome  or  value.  The  best  utility  is  subjectively  set 
to  one  with  the  worse  being  zero.  The  best  outcome  of  National  Survival  utility  would  be 
secure,  while  the  best  force  structure  cost  utility  would  be  the  cheapest  or  the  one-half 
MRC  force,  and  both  are  assigned  a  utility  of  one.  Here,  the  perturbations  only  affect  the 
outcomes  which  fall  between  the  best  and  worse  case.  This  is  due  to  how  one  and  zero 
are  assigned.  These  central  utilities  were  varied  by  plus  or  minus  0.1,  with  all  the  central 
outcomes  being  either  increased  or  decreased  at  the  same  time.  An  example  of  how 
Force  Structure  Cost  Utility  is  perturbed  is  shown  in  Table  3.9. 

The  utility  weight  coefficients  must  sum  to  one  in  order  to  meet  the  additive  utility 
function  requirements.  This  leads  to  a  forced  grouping  where  perturbing  one  coefficient 
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Initial  Force 

Force  Structure 

Force  Structure  Cost 

Perturbation  of  Force 

Structure 

Buildup 

Utility 

Structure  Cost  Utility 

Two  MRC 

0.0 

0.0 

One  MRC 

Two  MRC 

0.2 

0.2  +!-  Utility  Perturbation 

One  MRC 

0.4 

0.4  +!-  Utility  Perturbation 

One-half  MRC 

Two  MRC 

0.5 

0.5  +/-  Utility  Perturbation 

One-half  MRC 

One  MRC 

0.7 

0.7  +!-  UtilityPerturbation 

One-half  MRC 

1.0 

1.0 

Table  3.9  Perturbation  of  Force  Structure  Cost  Utility  (FSCU) 


requires  a  change  to  the  others.  The  original  utility  function  uses  the  fact  that  National 
Survival  Utility  is  twice  as  important  as  either  National  Interests  or  Force  Structure  Cost 
Utilities.  This  dependence  is  expressed  with  the  following  function: 

U(Su  rvival  Interests,  Cost)  =  (1  -  2C)  *  U(Survival)  +  C  *  U(lnterests)  +  C  *  U(Cost)  (3.3) 

The  coefficient  C  is  initially  set  to  0.25. 

The  influence  diagram  of  the  original  sensitivity  analysis  on  the  three  variable 
groups  is  shown  in  Figure  3.2.  There  are  three  value  nodes  used  as  perturbation  variables: 
Probability  Perturbation,  Utility  Perturbation,  and  the  Cost  Weight  perturbation.  These 
are  the  variables  which  are  actually  changed  across  the  ranges  previously  defined.  The 
other  variables  are  defined  in  tems  of  their  particular  perturbation  variable.  This  Influence 
Diagram  is  used  to  generate  the  tornado  diagram  found  in  Figure  3.3. 
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Force  Structure  -  3  variable  group  analysis 


Figure  3.2  Influence  Diagram  for  Three  Group  Perturbations 


The  tornado  diagram  presents  the  effects  of  perturbing  one  main  group  at  a  time. 
This  one-way  sensitivity  analysis  displays  the  effect  of  the  perturbations  on  the  overall 
expected  utility  and  any  changes  in  decision  policy.  The  shaded  areas  indicate  that  the 
optimal  alternative  has  changed  from  the  original  setting.  DPL^M  evaluates  the  model  at 
three  points;  the  original  base  case  and  two  endpoints  on  either  side  of  the  original.  The 
shaded  area  shows  only  that  the  change  in  policy  occuiTed  somewhere  between  the 
original  and  the  endpoint  evaluation. 

This  initial  group  analysis  gives  some  insight  into  the  effect  of  changing  the 
probability,  utility  or  utility  weight  parameters.  Perturbing  any  of  these  parameters  can 
lead  to  a  change  in  expected  utility  and/or  decision  policy.  The  objective  so  far  is  to  gain 
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Tornado  diagram  of  3  group  sensitivities 

0.77 


Overall  Utility  Function  Weight  Perturbations 


0.3/0.732 

Probability  Perturbation 

0.2/0.807 

1 

0.1  /  0.756 

Utility  Perturbation 

-0.1/0.81 

-0.1  /  0.766 

- ^ ^ ^ 

0.1  /  0.774 

- ^  T 

0.74  0.75  0.76  0.77  0.78  0.79  0.8 

Expected  Utility 


Figure  3.3  Tornado  Diagram  of  Three  Main  Groups  One-way  Sensitivity  Analysis 

insight  into  the  significant  variables.  From  here,  it  would  be  useful  to  narrow  the  field  to 
study  the  most  important  or  sensitive  variables.  At  this  point,  there  is  no  evidence  to 
suggest  throwing  any  of  the  groups  out.  Therefore,  the  number  of  groups  was  expanded 
by  dividing  them  into  1 1  subgroups 

Grouping  at  a  Lower  Level  of  Detail.  The  eleven  new  groups  were 
defined  by  using  the  preexisting  nodes  in  the  influence  diagram.  The  probability  group 
was  broken  down  according  to  the  uncertain  variables  in  the  chance  nodes.  The  utility 
group  was  broken  down  into  the  three  smaller  utility  groups;  national  survival  utility, 
national  interests  utility  and  force  structure  cost  utility.  The  overall  weight  coefficient 
group  could  not  be  looked  at  in  any  finer  detail,  so  it  remained  unchanged. 
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An  influence  diagram  with  value  nodes  identifying  the  eleven  group  perturbations 


is  shown  in  Figure  3.4. 


Figure  3.4  Influence  Diagram  11  groups  used  for  Sensitivity  Analysis 


Group  Variables  used  to  make  perturbations  for  sensitivity  analysis: 

National  Survival  threat  Intelligence  prediction  Probabilities  -  NSIP 

National  Survival  Threat  Probabilities  -  NSTP 

National  Survival  Probabilities  -  NSP 

National  Survival  Utility  -  NSU 

Force  Structure  Available  Probabilities  -  FSAP 

Cost  Weight  coefficient  to  overall  utility  function  -  CW 

Force  Structure  Cost  Utility  -  FSCU 

National  Interests  threat  Intelligence  prediction  Probabilities  -  NIIP 
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National  Interests  Threat  Probabilities  -  NITP 
National  Interests  Probabilities  -  NIP 
National  Interests  Utility  -  NIU 

One-way  Sensitivity  Analysis.  These  groups  were  perturbed  similar  to  the 
previous  groupings  to  create  a  tornado  diagram  shown  in  Appendix  E.  The  probability 
and  utility  perturbations  were  all  plus  and  minus  0.1  from  the  significant  element  in  each 
vector.  The  cost  weight  was  again  varied  from  0.2  to  0.3. 

The  top  four  groups  have  the  greatest  effect  on  expected  utility  for  the  range  of 
values  selected.  Seven  of  the  top  eight  indicate  at  least  one  change  in  decision  policy  at 
some  point  in  the  perturbation  range.  Remember,  this  figure  is  based  on  changing  one 
parameter  at  a  time  with  the  others  set  at  their  base  value.  This  gives  no  information  on 
how  the  decision  is  affected  if  more  than  one  group  is  varied. 

Two-way  Sensitivity  Analysis.  DPL's  rainbow  diagram  option  is  used  to  do 
two-way  sensitivity  analysis.  The  rainbow  diagram  runs  the  model  across  a  range  of 
settings  (up  to  21)  for  one  variable.  If  there  is  a  specified  relationship  between  two 
variables,  they  can  be  varied  at  the  same  time  by  defining  one  variable  in  teims  of  the 
other,  or  by  defining  both  variables  in  terms  of  a  third.  The  other  possibility  is  to  define  an 
array  with  settings  of  the  two  variables  and  running  the  model  at  each  setting  in  the  array 
(1:333-357).  The  best  option  for  this  model  was  to  use  a  rainbow  diagram  of  one  variable 
across  the  interval  -0.1  to  0.1  and  run  this  rainbow  diagram  eleven  times.  The  second 
variable  was  incremented  between  runs  to  establish  a  strategy  region  graph  seen  in  Table 
3. 10.  The  extraction  of  the  data  from  the  rainbow  diagrams  after  each  run  was  a  rather 
painstaking  process,  but  it  did  provide  the  means  to  gather  two-way  sensitivity 
information. 
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The  strategy  region  displays  the  optimal  decision  policy  across  the  desired  range  of 
interest.  The  line  that  splits  the  two  decision  strategies  does  not  lie  on  the  boundary  of 
values.  This  line  may  lie  anywhere  between  the  points  evaluated  in  the  0.02  wide  interval. 
This  level  of  accuracy  is  acceptable  in  most  cases,  since  the  strategy  region  will  show  the 
general  trend. 

FSCIJ 

-0.1  -0.08  -0.06  -0.04  -0.02  0  0.02  0.04  0.06  0.08  0.1 

-0.1 
-0.08 
-0.06 
-0.04 
-0.02 

NSTP  0 
0.02 
0.04 
0.06 
0.08 
0.1 

Table  3.10  Decision  Policy  Based  on  Perturbations  of  Force  Structure  Cost  Utility 
vs  National  Survival  Threat  Probability  (1  or  2  MRC  Force) 
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If  the  estimation  of  the  most  likely  values  of  the  uncertain  variables  is  near  the 
break  point  between  a  one  and  two  MRC  force,  the  optimal  decision  would  be  highly 
sensitive  to  the  actual  outcome  of  these  variables.  It  is  critical  to  realize  that  these  figures 
are  based  on  expected  outcomes  and  must  not  be  taken  literally.  In  other  words,  as  the 
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estimates  approach  the  bottom  left  corner  of  Table  3.10,  the  level  of  confidence  increases 
that  a  two  MRC  force  is  the  best  decision.  Another  approach  that  would  give  the  same 
results  would  be  to  use  the  actual  equations  generated  from  folding  back  the  decision  tree 
(3:121-136).  This  would  be  a  ambitious  undertaking,  considering  that  the  equation  for 
each  of  the  three  alternatives  would  require  29,439  multiplication's,  19,626  additions,  and 
81  comparisons.  This  is  based  on  a  symmetric  decision  tree  that  would  be  an  expanded 
version  of  Figure  3.5.  These  equations  would  have  to  be  entered  manually  into  a 
spreadsheet  and  compared  at  the  various  input  values  of  interest.  This  approach  would 
certainly  save  on  the  number  of  model  runs,  but  would  be  impractical  for  all  but  the 
simplest  of  models. 


FORCE  STRUCTURE  MODEL  DECISION  TREE 


Initial  National  National  Force  National  National  Force  i  m  +■  i 

Force  Survival  Interests  Structure  Survival  Interests  Structure  ,  National 

Structure  Intell  Intel!  Buildup  Threat  Threat  Available  buryiya! 


One- haft  MRC 
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'\^  LoW  ^  I 
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A 


One-Kd¥MRC 


V 


A, 


Ore-Half  MRC 


-c 


■/  Marginal 


Value 

Marginal  Marginal 

Value 
Intecure 


<] 


THE  FULL  EXPANDED  DECISION  TREE  CONSISTS 
OF  3''9  =  19,683  POSSIBLE  OUTCOME  BRANCHES 
AT  THE  FAR  RIGHT  OF  THE  TREE. 

THAT  IS  29,523  TOTAL  BRANCHES  OR  NODES. 
THIS  WOULD  CERTAINLY  BE  A  MESSY  DIAGRAM 
THAT  WOULD  UNDOUBTFULLY  CONFUSE  RATHER 
THAN  CLARIFY  THE  PROBLEM. 


Figure  3.5  Condensed  Decision  Tree  for  Force  Structure  Problem 
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Although  the  one-way  sensitivity  analysis  of  this  decision  model  is  fairly 
straightforward,  the  ability  to  perfonn  higher  level  analysis  is  quickly  restricted  by  a 
limited  number  of  tools  in  the  software.  This  is  where  there  is  room  for  improvement. 

Response  Surface  Methodology  Approach  to  Sensitivity  Analysis 

Response  Surface  Methodology  (RSM)  is  simply  a  method  of  estimating  a 
response  surface  through  experimental  design.  The  end  result  being  a  parsimonious  model 
that  closely  predicts  the  results  of  the  model  in  some  region  of  interest.  This  would 
certainly  aid  in  doing  sensitivity  analysis  on  the  model  if  we  could  accurately  portray  the 
response  surface  with  an  equation  that  doesn't  include  almost  50,000  arithmetic  operators. 

To  begin  this  portion  of  the  analysis,  the  eleven  groups  described  previously  were 
used.  Initially,  to  determine  if  this  would  work  at  all,  a  Plackett-Burman  design  was  used 
to  screen  these  eleven  groups  for  significant  factors.  This  required  a  12  run  design  that 
was  of  resolution  HI  (2: 162).  Therefore,  the  eleven  main  factors  would  not  be 
confounded  with  main  factors,  but  any  interactions  would  be  confounded  with  the  main 
factors.  The  main  factors  were  all  coded  to  a  range  of  1  to  -1  to  ease  the  programming 
and  analysis.  This  coding  was  done  using  the  standard  methods  shown  in  Equation  (3-4). 

XcODED  =  (XuNCODED  “  ^midPOINt)  !  ^HALF-WIDTH  (3-4) 

where  S^j^point  =  the  midpoint  of  the  interval  of  perturbation  (0  or  0.25) 

^HALF-WIDTH  =  the  half-width  of  the  interval  of  perturbation  (0. 1  or  0.05) 

Therefore,  an  Xuncoded  =  'O-l  would  transform  to  a  X^q^ed  =  -  l-O-  The  12 
runs  were  made  using  the  uncoded  values  in  DPL™.  The  coded  values  from  the  design 
matrix  were  regressed  against  the  expected  utility  response  Y. 
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The  original  Plackett-Burman  design  mati'ix  and  response  is  shown  in  Table  3.1 1. 


GROUP 

NSIP  NIIP  NSTP  NITP  NSP  NSU  NIP 

NIU 

FSAP 

FSCU 

CW  E(UTIL) 

FACTOR 

RUN 

I 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

Y 

1 

1 

1 

1 

-1 

1 

1 

1 

-1 

-1 

-1 

1 

-1 

0.835704 

2 

1 

-1 

1 

1 

-1 

1 

1 

1 

-1 

-1 

-1 

1 

0.700124 

3 

1 

1 

-1 

1 

1 

-1 

1 

1 

1 

-1 

-1 

-1 

0.8 

4 

1 

-1 

1 

-1 

1 

1 

-1 

1 

1 

1 

-1 

-1 

0.814776 

5 

1 

-1 

-1 

1 

-1 

1 

1 

-1 

1 

1 

1 

-1 

0.825202 

6 

1 

-1 

-1 

-1 

1 

-1 

1 

1 

-1 

1 

1 

1 

0.76432 

7 

1 

1 

-1 

-1 

-1 

1 

-1 

1 

1 

-1 

1 

1 

0.81264 

8 

1 

1 

1 

-1 

-1 

-1 

1 

-1 

1 

1 

-1 

1 

0.745588 

9 

1 

1 

1 

1 

-1 

-1 

-1 

1 

-1 

1 

1 

-1 

0.821544 

10 

1 

-1 

1 

1 

1 

-1 

-1 

-1 

1 

-1 

1 

1 

0.711047 

11 

1 

1 

-1 

1 

1 

1 

-1 

-1 

-1 

1 

-1 

1 

0.7 

12 

1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

0.8264 

Table  3.11  Initial  Plackett-Burman  Design  on  11  Group  Factors 


The  resulting  ANOVA  table  suggests  that  the  only  significant  factors  are  the 
National  Survival  Threat  Probability  (NSTP),  Force  Structure  Cost  Utility  (FSCU),  and 
the  Cost  Weight  (CW).  Also,  by  examining  the  intercept,  which  is  0.776614  to  the 
expected  value  of  0.769821  of  the  base  case,  it  seems  that  the  regression  line  will  hold 
across  the  entire  range.  This  suggests  that  a  linear  first  order  model  might  be  sufficient  to 
approximate  the  response  surface.  After  a  number  of  variations,  a  2^  full  factorial  design 
with  the  center  point  included  was  run  on  the  three  significant  factors.  The  equation  that 


3-18 


surfaced  is  listed  below  with  the  accompanying  ANOVA  Table  3.12. 


E(U)  =  0. 776614  -  0. 01629  *  NSTP  +  0. 018088  *  FSCU  -  0. 0403 1  *  CW 

-0.00691  *  NSTP  *  FSCU  +  0.00781FSCU  *  CW  (3-5) 


Regression  Statistics  Analysis  of  Variance 


Multiple  R 

0.996737 

df 

MS 

F 

R  Square 

0.993484 

Regress 

5 

0.01861 

0.003722 

91.47956 

Adjusted  R  Square 

0.982624 

Residual 

3 

0.00012 

4.07E-05 

Standard  Error 

0.006379 

Total 

8 

0.01873 

Observations  9 


Coef 

Std  Err 

t  Statistic 

P-value 

Intercept 

0.776614 

0.002126 

365.2378 

3.54E-18 

NSTP 

-0.01629 

0.002255 

-7.22163 

9.05E-05 

FSCU 

0.018088 

0.002255 

8.020193 

4.29E-05 

CW 

-0.04031 

0.002255 

-17.8756 

9.83E-08 

NSTP*FSCU 

-0.00691 

0.002255 

-3.06477 

0.01547 

FSCU*CW 

0.00781 

0.002255 

3.462943 

0.008531 

Table  3.12  Analysis  of  Variance  Table  for  regression  of: 
E(U)=bO+bl*NSTP+b2*FSCU+b3*CW+b4*NSTP*FSCU+b5*FSCU*CW 


The  equation  turns  out  to  be  quite  accurate  with  a  highly  significant  F-test  and  a  R 
square  and  adjusted  R  square  which  explains  much  of  the  data.  The  individual  factor  t- 
tests  also  show  a  high  level  of  confidence  with  a  p-value  below  0.02  in  all  cases.  This  first 
success  at  predicting  the  response  surface  of  the  force  model  leads  to  further  exploration 


Signif  F 

0.001775 
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of  it's  possible  applications.  At  this  point,  there  is  an  accurate  approximation  of  the 
optimal  expected  value  across  the  region  of  interest.  This  is  useful  for  the  prediction  of 
the  optimal  expected  value,  but  it  doesn't  define  any  changes  in  policy.  Without  the  ability 
to  predict  changes  in  the  optimal  policy,  the  results  are  not  veiy  useful. 

The  next  step  is  to  develop  an  equation  for  each  of  the  three  alternatives  and  use 
these  to  produce  strategy  region  graphs.  With  this  in  mind,  the  initial  Plackett-Burman 
design  runs  were  used  with  the  addition  of  the  expected  Utility  of  each  of  the  three 
alternatives.  The  first  regression  on  the  response  for  the  two  MRC  alternative  was  quite 
surprising.  The  only  significant  factor  was  that  of  Cost  Weight.  After  a  little  refining  of 
the  equation,  it  turns  out  that  we  can  perfectly  predict  the  expected  utility  of  the  two  MRC 
option.  The  expected  value  of  this  option  can  be  defined  with  Equation  (3-6). 

E(U),mrc  =0.75-0. 05  =<'CW  (3.6) 

Remember  that  these  equations  are  for  the  coded  values  of  the  input  variables.  The  coded 
value  CW  is  a  function  of  the  uncoded  cw  and  is  determined  by  Equation  (3-7). 

CW  =  (cw- 0.25)/ 0.05  (3.7) 

Therefore  a  value  of  0.2  would  be  coded  as  -1  in  the  design  matrix  and  so  on.  It  is  a 
simple  matter  to  reverse  the  process  to  decode  these  variables  and  use  the  regression 
equations. 

The  same  basic  procedure  was  used  to  identify  an  equation  for  the  expected  utility 
of  the  1  MRC  alternative.  However,  the  initial  Plackett-Burman  design  suggested  that 
there  were  four  groups  that  were  significant  to  approximate  this  response.  These 
significant  groups  were  NSTP,  NITP,  FSCU  and  CW.  In  order  to  accurately  examine 
these  factors  a  2^  full  factorial  design  was  run  using  all  four  main  factors  and  all 
combinations  of  interaction  terms.  The  follow-on  analysis  of  the  ANOVA  table  culled  the 
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field  to  a  linear  combination  of  the  four  factors  with  one  interaction  teiTn.  The  final 
approximating  Equation  (3-8)  and  the  associated  ANOVA  Table  3.13  are  found  below. 


E(U)imrc  =  0. 771733  -  0. 02365  *  NSTP  -  0.0156  *  NITP  +  0. 025  *  FSCU 

-0. 03726  *CW+0. 0058  *FSCU*CW  (3-8) 


Rearession  Statistics 

ANOVA 

Multiple  R 

0.995456 

df 

MS 

F 

SigF 

R  Square 

0.990932 

Regression 

5 

0.045458 

0.009092 

240.4038 

7.6E-11 

Adjusted  R  Square 

0.98681 

Residual 

11 

0.000416 

3.78E-05 

Standard  Error 

0.00615 

Total 

16 

0.045874 

Observations 

17 

Coef 

Std  Error 

tStat 

P-value 

Intercept 

0.771733 

0.001492 

517.4185 

1.76E-25 

NSTP 

-0.02365 

0.001537 

-15.3816 

8.75E-09 

NITP 

-0,0156 

0.001537 

-10.1473 

6.39E-07 

FSCU 

0.025 

0.001537 

16.26113 

4.86E-09 

CW 

-0.03726 

0.001537 

-24.2375 

6.74E-11 

FSCU*CW 

0.005 

0.001537 

3.252225 

0.007705 

Table  3.13  Analysis  of  Variance  Table  for  Regression  of  1  MRC  option: 
E(U)=bO+bl*NSTP+b2*NITP+b3*FSCU+b4*CW+b5*FSCU*CW 


The  results  show  that  we  can  explain  the  data  at  a  high  degree  of  confidence  with 
residuals  limited  to  the  third  or  fourth  decimal  place.  Although  this  equation  is  a  bit  more 
complex  than  that  of  the  two  MRC  option,  it  is  still  easily  applied  to  spreadsheet  analysis 
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and  the  development  of  strategy  regions.  Here  again  the  group  factors  agree  with  the 
overall  tornado  diagram  as  to  the  level  of  significance  and  relative  ranking.  The  next  step 
is  to  look  at  the  last  alternative  of  a  one-half  MRC  force. 

The  one-half  MRC  force  response  was  similarly  examined  against  the  Plackett- 
Burman  design.  In  this  case  there  was  evidence  that  three  group  factors  were  significant. 

These  significant  group  factors  were  NSTP,  FSAP,  and  CW.  These  factors  were  used  to 
create  a  2^  full  factorial  design,  which  was  run  using  the  DPL™  model.  The  resulting 
approximation  Equation  (3-8)  and  ANOVA  Table  3.14  are  shown  below. 

E(U)„2mrc  =  0-  724785  -  0. 03329  *  NSTP  -  0. 02035  *  FSAP  -  0. 02033  *  CW  +  0. 006069  *  NSTP  *  CW  (3.9) 

ANOVA 

df  SS  MS  F  SigF 
Regression  4  0.016463  0.004116  596.2669  8.4E-06 

Residual  4  2.76E-05  6.9E-06 

Total  8  0.016491 


Coef 

Std  Error 

t  Stat 

P-value 

Intercept 

0.724785 

0.000876 

827.6086 

1.28E-11 

NSTP 

-0.03329 

0.000929 

-35.8392 

3.62E-06 

FSAP 

-0.02035 

0.000929 

-21.905 

2.57E-05 

CW 

-0.02233 

0.000929 

-24.0438 

1.77E-05 

NSTP^CW 

0.006069 

0.000929 

6.533262 

0.002836 

Table  3.14  Analysis  of  Variance  for  Regression  of  1/2  MRC: 
E(U)=bO+bl*NSTP+b2*FSAP+b3*CW+b4*NSTP*CW 
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This  equation  is  also  quite  significant  with  good  explanatory  power.  Although  this 
alternative  does  not  become  a  optimal  choice  in  the  region  of  interest,  this  equation  can 
still  be  a  useful  tool  for  sensitivity  analysis. 


Using  Response  Surface  Approximations  to  Generate  Strategy  Regions. 

The  strategy  region  graph  presented  in  the  standard  sensitivity  analysis  section  is  a 
excellent  tool  for  presenting  two-way  analysis.  The  new  approach  is  to  use  the  three 
equations  generated  through  the  use  of  RSM  to  estimate  each  alternative's  expected 
utility. 


NSTP 


Strategy  Region  using  RSM  Equations 


—  E2(U)-E1(U)  =  0 


FSCU 


Figure  3.6  Decision  Policy  Based  on  Perturbations  of  FSCU  and  NSTP 

(1  or  2  MRC) 
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Here,  the  equations  are  evaluated  across  the  region  of  interest.  The  alternatives  are 
compared  to  generate  a  strategy  region  graph  like  that  shown  previously  in  Figure  3.4. 
The  result  shows  that  there  are  only  two  optimal  options  in  this  region  (1  and  2  MRC). 
Therefore,  the  expected  utility  for  the  one-half  MRC  force  is  not  a  factor  in  this  region. 
The  strategy  region  depicted  in  Figure  3.6  is  constructed  by  evaluating  E2mr(;;;(U)  - 
E2MRC(U)  =  0. 
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IV.  Analysis 


Initial  Solution 

The  initial  solution  to  the  uncertain  force  structure  model  corresponds  to  the 
maximum  expected  utility  across  the  force  structure  alternatives.  This  maximum  expected 
utility  does  little  to  explain  the  consequences  of  such  a  decision.  The  only  clear  result  is 
that  the  optimal  decision  policy  is  expected  to  give  the  best  overall  utility  as  defined  in  the 
utility  function. 

Using  the  original  utility  function  in  Equation  (3-3)  and  solving  the  influence 
diagram  yields  an  expected  utility  of  0.77.  This  result  is  based  on  using  an  initial  one  MRC 
force  structure  as  seen  in  the  decision  policy  attached  in  Appendix  A.  The  optimal  force 
structure  buildup  is  dependent  on  predicted  future  threats  and  is  shown  in  Table  4.1. 

Table  4.1  shows  that  a  buildup  to  a  two  MRC  force  is  needed  when  the  intelligence 
prediction  of  the  national  survival  threat  is  upgraded  to  high.  While,  a  one  MRC  force  is 
still  optimal  if  the  intelligence  prediction  of  the  national  survival  threat  is  low.  In  the  case 
where  the  national  survival  threat  is  predicted  to  be  medium,  the  predicted  threat  to 
national  interests  should  be  considered.  All  of  the  optimal  one  MRC  regions  were  equal  to 
the  one-half  MRC  expected  utilities  for  the  force  structure  buildup  decision.  This  is  due  to 
the  model  structure,  where  reductions  in  force  are  not  an  option.  Therefore,  the  one  MRC 
force  would  be  the  solution  in  these  areas. 

It  can  be  assumed  that  the  future  decision  to  buildup  the  force  structure  will  be 
continually  assessed.  In  this  case,  the  influence  diagram  would  be  restructured  and 
reevaluated  based  on  the  threat  predictions  at  that  time.  With  this  in  mind,  the  sensitivity 
analysis  is  focused  on  the  initial  force  structure  decision. 
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PREDICTED  THREAT  TO  NATIONAL  INTERESTS 


HIGH 

MEDIUM 

LOW 

HIGH 

TWO  MRC 

TWO  MRC 

TWO  MRC 

0.627 

0.658 

0.664 

PREDICTED 

THREAT  TO 

MEDIUM 

TWO  MRC 

ONE  MRC* 

ONE  MRC* 

NATIONAL 

0.743 

0.798 

0.811 

SURVIVAL 

LOW 

ONE  MRC* 

ONE  MRC* 

ONE  MRC* 

0.754 

0.816 

0.828 

Table  4.1  Optimal  Decision  Policy  for  the  Force  Structure  Buildup  Alternatives 
given  a  One  MRC  Initial  Force  Structure 
*  These  expected  utilities  were  tied  with  the  one-half  MRC  option 

Discussion  of  One-way  Sensitivity  Analysis 

The  objective  of  the  one-way  sensitivity  analysis  is  to  identify  significant  factors  and 
their  impact  on  the  optimal  policy.  The  first  technique  applied  to  generate  this  one-way 
analysis  was  the  tornado  diagram.  The  tornado  diagram  was  produced  in  DPL™,  which 
varied  one  factor  at  a  time  and  evaluated  the  model  at  the  base  case  and  the  endpoints. 

This  required  23  additional  runs  of  the  model  (the  endpoints  of  1 1  factors  plus  the  base 
case).  The  resulting  tornado  diagram,  shown  in  Appendix  E,  indicates  that  changes  in 
seven  of  the  eleven  factors  would  affect  the  optimal  policy. 

Cost  weight  had  the  greatest  impact  on  the  overall  expected  value,  but  did  not 
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impact  the  optimal  decision.  Therefore,  the  weights  of  the  three  individual  utility  functions 
are  not  critical  to  the  overall  decision  in  this  range  of  interest.  The  next  seven  variables 
prove  to  be  the  most  interesting,  changing  the  decision  policy  with  perturbations  in  the 
range  of  -0.1  to  0.1.  These  significant  variables  are  listed  here  in  descending  order  of 
impact  on  the  overall  utility:  FSCU,  NSTP,  NITP,  FSAP,  NIP,  NIU,  and  NIIP. 

It  is  interesting  to  note  that  NSP,  NSU,  and  NSIP  are  not  significant  in  the  range 
evaluated.  This  is  counterintuitive  when  considering  that  national  survival  utility  is 
weighted  twice  as  much  as  the  other  two  individual  utilities.  Apparently,  the  national 
survival  utility  and  national  survival  probabilities  could  not  be  changed  enough  to  make  an 
impact.  This  is  most  likely  due  to  the  perturbations  affecting  only  the  marginal  outcome, 
since  secure  and  insecure  are  assigned  utilities  of  1  and  0,  respectively.  National  interests 
utility  and  probabilities  (NIU,  NIP)  are  perturbed  similarly,  but  the  probability  of  the  acmal 
national  interests  threat  (NITP)  of  being  medium  or  high  was  considerably  more  than  for 
the  actual  national  survival  threat  (NSTP).  Table  3.1  shows  the  actual  threat  probabilities 
for  national  survival  and  national  interests.  Since  the  threat  is  less  likely  to  become  a  factor 
for  national  survival,  the  intelligence  prediction  of  national  survival  threat  probabilities 
(NSIP)  is  also  insignificant  in  this  range.  Therefore,  these  variables  would  be  dropped 
from  further  analysis  in  a  standard  sensitivity  analysis. 

It  is  important  to  realize  that  tornado  diagrams  show  changes  in  decision  policy 
(either  decision),  not  just  the  initial  decision.  This  fact  will  become  clear  when  comparing 
the  strategy  regions  at  the  end  of  this  chapter  with  the  tornado  diagram  .  The  analyst 
should  run  the  model  at  specific  points  of  interest  to  clarify  the  information  gained  from  the 
tornado  diagram.  This  will  require  some  additional  runs  of  the  model,  but  should  ensure 
that  the  correct  conclusions  are  drawn.  Also,  since  the  tornado  diagram  doesn't  specify 
where  the  decision  policy  changes,  it  is  necessary  to  systematically  narrow  the  range  to 
pinpoint  this  crossover.  Again  this  requires  a  number  of  additional  runs  of  the  model. 
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depending  on  the  accuracy  required. 

In  comparison  to  the  tornado  diagram,  the  response  surface  approximation  for  the 
overall  utility  required  only  twelve  runs.  The  twelve  runs  are  those  generated  from  the 
Plackett-Burman  design  shown  in  Table  3.1 1.  The  equation  generated  shows  that  the  most 

E(U)  =  0. 776614  -  0. 01629  *  NSTP  +  0. 018088  *  FSCU  -  0. 0403 1  *  CW 

-0. 0069 1  *  NSTP  *  FSCU  +  0. 0078 IFSCU  *  CW  (4-1) 

significant  factor  in  predicting  the  overall  expected  utility  is  the  Cost  Weight  (CW).  The 
top  three  variables  in  the  tornado  diagram  are  ranked  in  the  same  order  in  this  equation, 
providing  some  validation  of  the  method.  Here,  "significance"  refers  to  the  effect  that  a 
particular  factor  has  on  predicting  the  expected  utility  of  the  optimal  decision  policy. 
However,  this  equation  does  nothing  to  identify  that  policy.  Although  it  is  comforting  to 
know  that  it  is  easy  to  generate  a  parsimonious  model,  this  equation  does  little  to  help  in 
the  sensitivity  analysis.  The  fact  that  this  expected  utility  is  the  maximum  of  the  three 
alternative's  expected  utilities  provides  access  to  more  in-depth  insight.  The  capability  to 
retrieve  the  alternative's  expected  values  is  made  possible  by  DPL^^.  The  expected 
utility  of  each  of  the  three  alternatives  is  found  in  the  Decision  Policy  output  as  seen  in 
Appendix  A  (1:306-309). 

The  three  alternative's  responses  were  gathered  for  all  of  the  12  runs  plus  the 
original  base  case  and  regressed  one  at  a  time  against  the  design  matrix.  This  data,  along 
with  the  augmented  Plackett-Burman  design  matrix  is  attached  in  Appendix  B.  As 
discussed  in  Chapter  III,  the  resulting  equations  are  highly  significant  in  a  statistical  sense. 
The  equations  are  listed  below  along  with  a  ranked  list  of  significant  factors  to  aid  the 
reader  in  the  following  analysis.  These  equations  are  still  in  the  coded  variable  form. 
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E(U)2mrC  =  0.75  -  0.05  *  CW 
Significant  factors: 

-  Cost  Weight  (CW) 

E(U)imrc  =  0. 77 1733  -  0. 02365  *  NSTP  -  0. 0156  *  NITP  +  0. 025  *  FSCU 
-0. 03726  *  CW  +  0. 0058  *  FSCU  *  CW 

Significant  factors: 

-  Cost  Weight  (CW) 

-  Force  Structure  Cost  Utility  (FSCU) 

-  National  Survival  Threat  Probability  (NSTP) 

-  National  Interests  Threat  Probability  (NITP) 

E(U),/2mrc  =  0-  724785  -  0.03329  *  NSTP  -  0.02035  *  FSAP  -  0.02033  *  CW  +  0.006069  *  NSTP  *  CW 
Significant  factors: 

-  National  Survival  Threat  Probability  (NSTP) 

-  Force  Structure  Available  Probability  (FSAP) 

-  Cost  Weight  (CW) 

It  is  a  simple  exercise  to  plug  in  the  variables  at  the  base  case,  where  all  of  our 
coded  variables  are  zero.  From  this,  it  is  established  that  the  maximum  of  the  three 
alternatives  is  the  E(U)i|\/|rq.  The  one-way  sensitivity  analysis  can  then  be  accomplished 

by  varying  one  factor  at  a  time  and  comparing  the  three  expected  utilities.  This  point  of 
view  would  indicate  that  there  are  only  five  significant  variables:  CW,  NSTP,  FSCU, 

FSAP,  and  NITP.  Obviously,  there  is  some  information  lost  at  this  point  due  to  the 
approximation.  The  question  is  how  much.  Upon  closer  evaluation  of  the  tornado 
diagram,  there  are  only  two  variables  that  affect  the  initial  force  stmcture  decision.  They 
are  FSCU  and  NSTP.  The  rest  of  the  variables  that  show  a  change  in  decision  policy  affect 
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only  the  force  structure  buildup.  With  this  in  mind,  the  information  lost  may  not  be  as 
critical  as  it  initially  appears.  In  this  sense,  our  coefficients  of  regression  have  picked  out 
the  most  significant  factors  in  predicting  the  optimal  initial  force  structure. 

Table  4.2  shows  the  results  of  comparing  the  three  alternative's  RSM  equations 
when  the  factors  are  perturbed  one  at  a  time.  The  calculations  are  performed  in  a 
spreadsheet  by  evaluating: 


Maximum  (E(U)2mrc  •  E{U)imrc  -  E(U)i/2mrc  )  (4-2) 


This  calculation  is  accomplished  with  logical  statements  producing  either  2,  1,  or  0.5  as  the 
output  at  each  factor  setting.  The  results  are  the  same  as  the  tornado  diagram.  The  only 
two  factors  which  change  the  initial  force  stmcture  decision  are  NSTP  and  FSCU.  Upon 
further  study  of  the  tornado  diagram,  it  was  found  that  the  decision  policy  change-over 
point  predicted  by  the  approximations  agree  to  within  0.01  of  the  factor  perturbation.  This 
level  of  accuracy  is  far  beyond  that  required  for  the  purpose  of  sensitivity  analysis  in  this 
model.  The  cost,  in  terms  of  model  runs,  is  the  12  Plackett-Burman  design  runs  plus  the 
2^  and  2^  full  factorial  design  runs.  These  36  runs  exceed  the  23  runs  needed  for  a  basic 


Range  of  perturbation 


FACTOR 


cw 

others 

0.2 

-0.1 

0.21 

-0.08 

0.22 

-0.06 

0.23 

-0.04 

0.24 

-0.02 

0.25 

0 

0.26 

0.02 

0.27 

0.04 

0.28 

0.06 

0.29 

0.08 

0.3 

0.1 

CW 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

NSTP 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2 

FSCU 

2 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

NITP 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

FSAP 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

Table  4.2  One-way  Sensitivity  Analysis  using  RSM  equations 
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tornado  diagram,  but  the  tornado  diagram  analysis  required  an  additional  100  or  more  runs 
to  clarify  the  results.  RSM's  fmgal  use  of  model  runs  has  only  begun  to  be  identified,  since 
no  further  runs  are  required  to  utilize  the  RSM  analysis.  The  RSM  equations  can  be  used 
in  the  current  form  to  predict  a  response  to  any  number  of  changes  in  the  parameters  across 
the  evaluated  design  region.  The  usefulness  of  the  approximations  formulated  using  RSM 
are  seen  again  in  the  two-way  analysis  included  in  the  next  section. 

The  analysis  of  varying  one  factor  at  a  time  indicates  that  there  are  only  two  factors 
which  effect  the  optimal  initial  force  structure  decision  in  the  region  of  interest.  These 
factors  are  the  force  structure  cost  utility  (FSCU  -  Table  3.6)  and  the  national  survival 
threat  probability  (NSTP  -  Table  3.1).  The  initial  force  required  is  increased  (1  to  2  MRC 
force),  when  the  force  structure  cost  utilities  are  decreased.  Remember,  only  the  utilities 
between  0  and  1  are  perturbed.  In  essence,  the  cost  of  starting  with  a  force  less  than  2 
MRC  and  then  building  up  has  gone  up  when  you  decrease  FSCU.  Since  these  option's 
utilities  are  getting  closer  to  zero  (the  most  expensive  case),  there  is  less  difference  in  cost 
between  a  2  MRC  force  and  a  lesser  force. 

On  the  other  hand,  positive  perturbations  in  the  actual  national  survival  threat 
probability  (NSTP)  shift  the  optimal  initial  force  structure  decision  from  a  1  to  a  2  MRC 
force.  This  is  sensible,  since  there  is  a  increasing  probability  of  the  national  survival  threat 
being  high  with  such  a  perturbation.  Individually,  the  effects  of  changing  these  two 
significant  factors  make  sense,  but  what  about  changes  in  both? 


Discussion  of  Two-way  Sensitivity  Analysis 

Standard  sensitivity  analysis  leads  to  an  examination  of  two-way  perturbations. 

The  obvious  factors  of  interest  are  those  that  proved  significant  in  the  one-way  analysis. 

As  discussed  in  Chapter  III,  DPL™  rainbow  diagrams  are  used  to  collect  the  data  required 
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for  the  two-way  sensitivity  analysis.  An  example  of  the  rainbow  diagram  is  attached  in 
Appendix  C.  This  rainbow  diagram  required  eleven  runs  of  the  model.  The  model  was 
evaluated  at  eleven  intervals  of  FSCU,  ranging  from  -0.1  to  0.1.  NSTP  was  set  at  0.1  for 
these  runs.  It  would  take  eleven  such  rainbow  diagrams  to  complete  a  strategy  region  as 
seen  in  Table  3.10.  The  total  cost,  in  terms  of  model  runs,  is  121  per  strategy  region.  In 
addition,  the  sh'ategy  region  must  be  generated  by  pulling  the  data  off  of  each  of  the  eleven 
rainbow  diagrams.  This  could  amount  to  a  lot  of  runs  and  time  if  there  were  a  number  of 
significant  factors  involved.  In  the  case  where  there  are  five  significant  factors,  it  would 
take: 

5!/  (2!*  (5  -  2)!)  =  10  strategy  regions 

These  ten  strategy  regions  would  demand  1210  model  runs  plus  the  time  spent 
pulling  1210  data  points  off  of  the  rainbow  diagrams.  Another  disadvantage  of  using  the 
rainbow  diagram  is  that  the  decision  change  is  shown,  but  the  actual  decision  is  not 
specifically  identified.  This  requires  model  runs  to  evaluate  what  the  decision  has  changed 
to.  At  this  point  in  the  analysis,  the  initial  method  required  nearly  250  model  runs  as 
compared  to  RSM's  36.  Obviously,  RSM  can  lead  to  a  significant  savings  in  time  and 
model  runs.  The  next  step  is  to  figure  out  the  cost  of  this  reduction,  in  terms  of  accuracy. 

There  were  several  strategy  regions  generated  with  both  methods  for  comparison. 
The  results  were  similar  from  a  accuracy  viewpoint.  The  RSM  method  was  accurate  to 
within  0.005  of  expected  value  results  and  agreed  with  the  rainbow  diagram  based  strategy 
regions  plus  or  minus  0.01  of  the  decision  crossover-point.  These  strategy  regions  would 
be  used  to  define  trends  and  to  locate  areas  were  the  decision  policy  is  sensitive  to  changes 
in  two  variables.  Therefore,  this  accuracy  is  sufficient  in  such  sensitivity  analysis. 
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Overall,  the  standard  sensitivity  analysis  is  more  accurate,  but  requires  an  order  of 
magnitude  more  runs  of  the  model.  The  efficiency  of  the  RSM  technique  would  be  more 
and  more  pronounced  as  the  model  increases  in  complexity. 
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V.  Conclusion  and  Recommendations 


The  creation  of  low  cost  microcomputer  software  packages,  such  as  DPL™^  has 
opened  the  doors  of  decision  analysis  to  new  users.  This  software  provides  the  analyst  and 
decision-maker  an  easily  accessible  and  flexible  tool  to  work  with.  DPL^M  furnishes  a 
user-friendly  interface,  aesthetically  pleasing  output  and  powerful  computational  features. 
Foremost  is  the  ability  to  formulate,  display  and  solve  Influence  Diagrams.  Influence 
Diagrams  are  an  excellent  tool  for  model  formulation,  but  just  as  importantly,  they  are 
robust  communication  tools.  They  are  unmatched  in  the  ability  to  convey  the  analyst's 
understanding  of  the  problem  at  hand  without  the  confusing  detail  of  other  methods.  And 
after  the  correct  problem  has  been  modeled,  the  solution  is  close  at  hand. 

Once  the  initial  solution  is  found,  sensitivity  analysis  becomes  the  focus  of  the 
analysis.  It  is  here,  that  the  standard  sensitivity  analysis  options  become  restrictive. 
DPL's^M  primary  sensitivity  tool  is  the  tornado  diagram.  The  tornado  diagram  is  an 
excellent  communication  tool  to  present  significant  factors  and  their  effects,  when 
perturbed  one  at  a  time.  A  problem  with  the  tornado  diagram's  created  in  DPL™^  is  the 
limited  capability  to  identify  specific  data  on  the  factors  effects.  However,  this  data  is 
available  with  additional  runs  of  the  model.  This  requirement  for  extra  clarifying 
information  became  a  trend  and  will  certainly  impact  the  analysis  of  any  substantial  decision 
model.  Beyond  the  tornado  diagram,  the  software  does  little  to  extend  sensitivity  analysis. 

At  this  point,  many  problems  require  some  two-way  sensitivity  analysis.  There  are 
three  options  that  might  be  considered  standard  for  two-way  analysis  in  decision  problems. 
The  first  would  be  to  run  the  model  at  all  of  the  possible  combinations  of  the  significant 
factors.  The  DPL™  application  of  this  would  include  building  an  array  of  the  possible 
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settings  and  programming  the  software  to  run  the  model  at  each  setting.  Obviously,  this 
becomes  a  considerable  task  for  many  models.  Next,  a  good  method  for  simple  problems, 
is  to  create  the  equation  produced  by  folding  back  a  decision  free  and  evaluating  it  at  the 
settings  of  interest.  This  method  requires  no  further  runs  of  the  model  and  produces 
perfect  results.  Here,  the  problem  lies  in  a  almost  infeasible  amount  of  time  to  produce 
such  an  equation.  As  mentioned  in  Chapter  III,  the  three  equations  in  the  force  structure 
problem  would  require  almost  50,000  arithmetic  operators  each.  Even  if  one  could  type  50 
words  (variables  plus  the  operators)  per  minute,  it  would  take  over  16  hours  to  type  in 
each  equation  into  a  spreadsheet.  There  must  be  a  better  way  and  there  is.  The  technique 
chosen  for  this  analysis  is  not  standard,  but  took  advantage  of  the  options  available  in  the 
software.  This  technique  used  the  rainbow  diagrams  as  discussed  in  Chapter  III  to  run  the 
model  over  the  desired  range  of  variables  to  compare  two  factor  effects.  Again,  this  could 
be  a  substantial  undertaking  for  complex  models  with  many  significant  factors  or  when  the 
analysis  requires  a  greater  level  of  accuracy. 

Overall,  the  options  discussed  above  require  many  additional  model  runs  and/or 
significantly  large,  sometimes  infeasible,  time  demands.  The  recommendation  here  is  to  use 
the  software  for  the  analysis  up  to  the  tornado  diagram.  After  which,  response  surface 
methodology  is  highly  effective. 

The  estimating  equations  produced  by  RSM  were  highly  explanatory,  which  is 
stunning  considering  only  36  model  mns  are  needed  to  produce  them.  Using  only  one 
strategy  region,  this  yields  a  overall  reduction  in  computer  mns  of  93%.  The  reduction  is 
potentially  much  higher,  considering  the  bulk  of  the  additional  runs  were  needed  for 
clarification  of  results  and  the  development  of  only  one  strategy  region.  The  cost  of  this 
reduction  is  in  an  acceptable  loss  of  accuracy.  The  incredible  savings  in  runs  leaves  ample 
room  to  make  a  small  number  of  mns  for  verification  of  the  results.  In  the  end,  the  RSM 
technique  proved  to  be  a  highly  successful  venture. 
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Overall,  RSM  provided  an  effective  and  efficient  sensitivity  tool  for  this  decision 
analysis  problem.  It  limited  the  model  runs  to  a  minimum  and  offers  highly  accurate  results 
over  the  design  region.  In  addition,  the  use  of  the  RSM  equations  facilitates  the 
construction  of  numerous  strategy  regions  with  no  additional  model  runs  required.  Finally, 
the  regression  coefficients  yield  insights  previously  unavailable  from  standard  sensitivity 
analysis. 

This  study  has  shown  that  response  surface  methodology  can  be  used  effectively  in 
the  sensitivity  analysis  of  decision  analysis  problems.  More  importantly,  RSM  significantly 
reduces  the  time  and  resources  required  to  produce  sensitivity  information.  This  savings  in 
time  is  well  worth  the  loss  of  accuracy  and  affords  a  great  deal  more  insight  into  the 
problem  at  hand.  This  study  suggests  the  potential  for  wide  use  of  RSM  in  the  field  of 
Decision  Analysis. 

Suggestions  for  Further  Study 

As  with  any  thesis,  time  constraints  limit  the  effort  much  sooner  than  potential 
research  areas.  There  are  three  areas  that  show  the  potential  for  further  study,  but  could 
not  be  accomplished  in  the  time  allowed.  The  first  involves  the  measurement  of  uncertainty 
in  terms  of  the  optimal  decision  policy.  There  needs  to  be  a  quantifiable  measure  of  risk, 
involving  the  potential  for  failure,  even  though  the  optimal  decision  was  made.  It  would 
seem  that  the  probability  of  the  future  threat  exceeding  the  force  stmcture  recommended 
can  be  calculated.  This  would  involve  backing  out  the  probability  of  actual  threats  and  the 
intelligence  prediction  given  these  threats.  It  would  also  have  to  take  into  account  the 
buildup  force  available  to  meet  an  upgraded  threat,  if  required.  The  resulting  measurement 
of  risk  would  aid  the  decision-maker  in  drawing  conclusions  and  making  future  evaluations 
of  the  value  of  information. 
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Next,  the  sensitivity  of  the  second  decision  on  the  force  structure  buildup  could  be 
analyzed  in  a  similar-  fashion.  This  would  include  RSM  equations  on  these  nine  possibilities 
and  how  they  react  to  perturbations  in  the  input  variables.  This  should  give  the  analyst  the 
tools  to  evolve  strategy  regions  that  are  broken  down  into  subdivisions  of  the  initial 
decision.  Again,  the  decision-maker  would  benefit  from  the  additional  information  both 
now  and  for  future  evaluations  of  the  model. 

Last  of  all,  three-way  sensitivity  information  would  be  easy  to  calculate  from  the 
RSM  equations,  but  difficult  to  present  and  interpret.  There  may  be  a  need  to  come  up 
with  a  three  dimensional  form  of  the  strategy  region  including  three  input  variables. 
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APPENDIX  A;  Decision  Policy  for  the  Initial  Solution 


Figure  A.l  Decision  Policy  for  Initial  Solution 
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APPENDIX  B;  Plackett-Burman  Design  used  in  RSM  Analysis 
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Table  B.l  The  Three  Force  Options  Expected  Utility  Response  vs  Plackett-Burman  Design 


Force  Structure  Cost  Utility  (FSCU)  perturbation 

Figure  C.  1  Example  of  Rainbow  diagram  Used  to  Create  Strategy  Regions 

C-1 


APPENDIX  D;  Strategy  Region  Comparison  for  Model  vs  RSM 
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APPENDIX  E:  Tornado  Diagram  of  1 1  Eactor  Groups 


Tornado  Diagram  of  11  group  Sensitivities 


one-way  Analysis 
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Tornado  Diagram  of  11  group  Sensitivities 
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Expected  Utility 


Figure  E.l  Tornado  Diagram  of  11  Groups  Perturbed  One  at  a  Time 
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